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(57) Abstract: The present invention provides apparatus and method for obtaining high dynamic range images using a low dynamic 
range image sensor. The image of a scene is captured with an image sensor using a spatially varying exposure function. The spatially 
varying exposure function can be implemented using an optical mask with a fixed spatial attenuation pattern, or using a controllable 
attenuation cell array. The captured image after calibration to correct for nonlinear response of the image sensor, is then normalized 
with respect to the exposure function. Where the exposure function is a fixed spatial attenuation pattern, the normalized image is 
interpolated to account for pixels that are saturated orb ln ck me d. Where the exposure function is implemented using a controllable 
attenuation cell array, the exposure function is adjusted to minimize the number of saturated or blackened pixels within an area of 
pixel array exposed through a cell, to maximize the signal-to-noise ratio of the average pixel value, and to maximize the spatial 
smoothness of the exposure function. Also disclosed is a technique for detailed preserving dynamic range compression of a high 
dynamic range image for display on a low dynamic range display device. The technique entails applying a smoothing filter to the 
exposure function used to obtain the high dynamic range image, and then reapplying the smoothed exposure function to the high 
dynamic range image. Also disclosed is a method for obtaining high dynamic range images by exposing photographic film using a 
spatially varying exposure function. The exposed film is then developed and scanned to derive an electronic image winch is aligned 
with the exposure function and processed in the same manner as images captured electronically using a fixed exposure function. 
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APPARATUS AND METHOD FOR HIGH DYNAMIC RANGE IMAGING 
USING SPATIALLY VARYING EXPOSURES 

BACKGROUND OF INVENTION 

The present invention relates to apparatus and method for capturing an image 
of a scene, and, more particularly, to apparatus and method for capturing a high 
dynamic range image using a low dynamic range image sensor 

Virtually any real world scene produces a very large range of brightness 
values. In contrast, known image sensing devices have very limited dynamic ranges. 
For example, it is typical for a video sensor to produce 8-bits or less of grey-level or 
color information. In the case of grey-scale images, 8-bits provide only 256 discrete 
grey levels, which is not sufficient to capture the fine details of most real life scenes. 

A known solution to the problem of capturing high dynamic range images with 
a low dynamic range image sensor is to take multiple image measurements for each 
local scene area while varying the exposure to light from the scene. Such exposure 
variation is typically accomplished by sequentially taking multiple images of the scene 
with different exposures and then combining the multiple images into a single high 
dynamic range image. Temporal exposure variation techniques for enlarging the 
dynamic range in imaging a scene may be found for example in: U.S. Patent 
5,420,635 to M. Konishi et al., issued May 30, 1995; U.S. Patent No. 5,455,621 to A. 
Morimura, issued October 3, 1995; U.S. Patent No. 5,801,773 to E. Ikeda, issued 
September 1, 1998; U.S. Patent No. 5,638,118 to K. Takahashi et al., issued June 10, 
1997; U.S. Patent No. 5,309,243 to Y.T. Tsai, issued May 3, 1994; Mann and Picard, 
"Being 'Undigital' with Digital Cameras: Extending Dynamic Range by Combining 
Differently Exposed Pictures " Proceedings of IST's 48 th Annual Conference, pp. 422- 
428, May 1995; Debevec and Malik, Recording High Dynamic Range Radiance 
Maps from Photographs," Proceedings of ACM SIGGRAPH, 1997, pp. 369-378, 
August 1997; and T. Misunaga and S. Nayar, "Radiometric Self Calibration," IEEE 
Conference on Computer Vision and Pattern Recognition (CVPR 99), June 1999. 
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However, techniques that require acquiring multiple images while temporally 
changing the exposure have the fundamental problem in that variations in the scene 
may take place between exposure changes. In other words, these techniques are useful 
only for static scenes where the scene radiance values stay constant. Moreover, 
5 between exposure changes the position and orientation of the imaging device and its 
components must remain constant. Finally, because a greater time is needed to 
sequentially capture all the required images, the temporal exposure variation 
techniques are not suitable for real time applications. 

Another known solution to the problem of capturing high dynamic range 

10 images with a low dynamic range image sensor is to simultaneously capture multiple 
images of a scene using different exposures. Such a technique is disclosed, for 
example, in Yamada et al., "Effectiveness of Video Camera Dynamic Range 
Expansion for Lame Detection," Proceedings of the IEEE Conference on Intelligent 
Transportation Systems, 1997. Typically, two optically aligned CCD light-sensing 

1 5 arrays are used to simultaneously capture the same image of a scene with different 
exposures. Light from the scene is divided by a beam splitter and directed to both 
CCD light-sensing arrays. The two captured images are combined into one high 
dynamic range image by a post processor. This technique has the disadvantage of 
requiring complex and expensive optics, and capturing images with more than two 

20 different exposures becomes difficult. 

Efforts have been made to increase the dynamic range of charge-couple 
imaging devices. Published Japanese patent application No. 59,217,358 of M. 
Murakoshi describes using two or more charge coupled device (CCD) light-sensing 
cells for each pixel of the imaging device. Each of the light-sensing cells of a pixel 

25 have different photo sensitivities so that some cells will take longer to reach saturation 
than others when exposed to light from a scene. In this manner, when the store 
photogenerated charge in all of the light-sensing cells of a pixel are combined, the 
dynamic range of the pixel is effectively increased. However, the Murakoshi 
reference does not address the problem of capturing high dynamic range images using 

30 a low dynamic range image sensor. 
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U.S. Patent No. 4,590,367 to J. Ross et al. discloses an arrangement for 
expanding the dynamic range of optical devices by using an electrically controlled 
light modulator adjacent to the light-sensitive area of an optical device to reduce the 
brightness of incident light from an imaged optical scene so as to not exceed the 

5 dynamic range or the light-sensitive area. The light modulator of the Ross et al. 

reference has individual pixel control of light amplification or attenuation in response 
to individual control signals. The detected level of light intensity emerging from each 
pixel of the modulator is then used to develop the control signals to adjust the 
amplification or attenuation of each pixel of the modulator to bring the brightness 

10 level into the detector's rated dynamic range. Thus, wide input light intensity 
dynamic range is reduced to a narrow dynamic range on a pixel-by-pixel basis. 
However, the apparatus and method of the Ross et al. reference is aimed at 
simplifying three-dimensional measurement using projected light, and there is nothing 
in the reference on how to capture a high dynamic range image. 

1 5 Another problem of known systems for capturing high dynamic range images 

is the display of such images using low dynamic range displays. Most commercially 
available displays, such as video monitors, televisions and computer displays, have 
low dynamic ranges. Hence, after the high dynamic range image is obtained, one 
needs to use a mapping method to display the image on a low dynamic range display 

20 while preserving the pertinent visual information. In other words, the dynamic range 
of the captured image must be compressed while preserving visual details of the 
scene. 

A known technique for compressing the dynamic range of a captured image is 
tone curve (or response curve) reproduction, in which a high dynamic range color 
25 scale is mapped into a low dynamic range color scale using an appropriate tone curve. 
Logarithmic scale conversion and gamma correction are examples of this kind of 
compression. However, tone curve reproduction compression does not preserve 
visual details of the scene. 

A more sophisticated tone curve reproduction technique is histogram 
30 equalization, which creates a tone curve by integrating the color histogram of the 
image. Histogram equalization results in a more even redistribution of the colors in 
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the image over the space of colors, but this technique cannot sufficiently preserve the 
detail of an image which has several local areas with very bright (or very dark) pixels, 
where such local areas together may span the complete range of grey levels. Hence, 
details within individual areas are not enhanced by the histogram equalization process. 
5 Another technique for compressing the dynamic range of a captured image is 

disclosed in Pattanaik et al., "A Multiscale Model of Adaptation and Spatial Vision 
for Realistic Image Display," SIGGRAPH 98 Proceedings, pp. 287-298, 1998. This 
technique divides the image data into several frequency component images, for 
example, in a Laplacian pyramid. The image is then re-composed after each 

1 0 frequency component has been modulated in a different manner. An advantage of this 
technique is that absolute pixel values do not contribute so much, unlike histogram 
equalization. However, this technique has the drawback of requiring large amounts of 
memory and extensive computations to obtain the frequency component images. 
Accordingly, there exists a need for an apparatus and method for capturing 

1 5 high dynamic range images using a relatively low dynamic range image sensor and for 
detail preserving compressing the captured high dynamic range image for display by 
relatively low dynamic range display device, which overcomes the problems of the 
prior art as discussed above. 



SUMMARY OF THE INVENTION 
20 In accordance with the present invention, there is provided a system for high 

dynamic range imaging comprising an image sensor having an array of light-sensing 
elements. The image sensor senses the image of a scene and provides corresponding 
pixel values representing the light intensities impinging on each of the light-sensing 
elements thereof. The image sensor provides a saturated pixel value when the 
25 intensity of light impinging on a corresponding light-sensing element is greater than a 
first threshold value and provides a blackened pixel value when the intensity of light 
impinging on a corresponding light-sensing element is below a second threshold level 
The image sensor has a low dynamic range relative to the range of light intensities in 
the optical image of the scene. 
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The system includes a mask interposed between the scene and the image 
sensor. The mask has a multiplicity of light transmitting cells, each controlling the 
exposure of a respective one or more of the light-sensing elements of the image sensor 
to light from the scene. Each of the light-sensing elements having a corresponding 
5 exposure value determined by the transparency of the cell through which light 
impinging on the light-sensing element passes. 

The system further includes a first memory storing exposure values 
corresponding to the light-sensing elements and an image processor coupled to the 
image sensor for receiving the pixel values provided thereby and coupled to the first 
1 0 memory for receiving the exposure values corresponding to the light-sensing 

elements. The image processor has a normalizer for mapping the pixel values by a 
function of the exposure values to derive corresponding normalized pixel values. An 
example of such mapping is to divide each of the pixel values by a respective 
exposure value. 

1 5 According to a first exemplary embodiment of the present invention, the 

exposure values corresponding to the light-sensing elements are fixed and the image 
processor further includes an interpolator for interpolating the normalized pixel values 
to derive interpolated pixel values at respective positions in a second array 
overlapping the array of light-sensing elements. The interpolation process may omit 

20 normalized pixel values that correspond to saturated pixel values or blackened pixel 
values. 

Where the response function of the image sensor is not linear, the system may 
include a second memory for storing the response function of the image sensor, and 
the image processor is coupled to the second memory for receiving the response 

25 function and further includes calibration means for linearizing the pixel values 

provided by the image sensor in accordance with the response function before the 
pixel values are normalized by the normalizer. 

According to a second exemplary embodiment of the present invention, the 
exposure value of each cell of the mask is alterable by applying a respective exposure 

30 control signal to the mask. The system includes a mask controller coupled to the 

image sensor for receiving the pixel values provided thereby and for computing, and 
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applying to the mask, exposure control signals for minimisin g the number of saturated 
pixel values and blackened pixel values from the corresponding area of the image 
sensor, and providing exposure values corresponding to the applied exposure control 
signals. The nonnalizer then normalizes each of the pixel values from the image 
sensor to derive normalized pixel values which represent a radiance image of the 
scene. 

The dynamic range of the captured high dynamic range image is compressed to 
allow viewing on a display having a relatively low dynamic range by providing a 
smoother coupled to an exposure pattern memory for receiving exposure values stored 
therein and applying a smoothing filter to the exposure values to derive smooth 
exposure values corresponding to the normalized pixel values. Also provided is an 
exposure reapplicator coupled to the nonnalizer of the image processor for receiving 
the normalized pixel values and for multiplying each of the normalized pixel values 
with a corresponding smoothed exposure value to derive monitor image pixel values 
of an image having compressed dynamic range but without substantial loss of image 
detail. 

In an alternative implementation of the first exemplary embodiment of the 
present invention, a mask is not used but the array of light-sensing elements of the 
image sensor has a spatially varying pattern of photosensitivities to incident light and 
corresponding photosensitivity values indicative of the respective photosensitivities. 
The photosensitivity values are stored in the first memory and are used by the 
nonnalizer in conjunction with respective pixel values from the image sensor to derive 
corresponding normalized pixel values, and there is provided an interpolator for 
interpolating the normalized pixel values to derive interpolated pixel values at 
positions in a second array overlapping the array of light-sensing elements. 

In accordance with another aspect of the present invention, there is provided a 
method for high dynamic range imaging comprising the steps of exposing an image 
sensor having an array of light-sensing elements to an image of a scene using a 
spatially varying exposure. The image sensor senses the image and provides 
corresponding pixel values representing light intensities impinging on respective ones 
of the light sensing elements. The image sensor provides a saturated pixel value when 
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the intensity of light impinging a respective light-sensing element is greater than the 
first threshold level and providing a blackened pixel value when the intensity of the 
light impinging on a respective light-sensing element is below a second threshold 
level. The image sensor having a low dynamic range relative to the range of light 
5 intensities in the image of the scene. The method further comprises the step of 

normalizing the pixel values provided by the image sensor using respective exposure 
values to derive corresponding normalized pixel values. 

According to one implementation of the method, the step of exposing the 
image sensor using a spatially varying exposure includes the step of using a mask 

1 0 having a multiplicity of light-transmitting cells each controlling the exposure of a 

respective one or more of the light-sensing elements to light from the scene. Each of 
the light-sensing elements having a corresponding exposure value indicative of the 
transparency of the cells through which light impinging on the light-sensing element 
passes, and the step of normalizing the pixel values provided by the image sensor 

1 5 includes the step of mapping the pixel values by a function of the exposure values. 

In another implementation of the method, the exposure values corresponding 
to the light-sensing elements are fixed, and the method further comprises interpolating 
the normalized pixel values to derive interpolated pixel values at respective positions 
of a second array overlapping the array of the light-sensing elements. 

20 In a further implementation, the method comprises the step of calibrating the 

pixel values provided by the image sensor according to a response function of the 
image sensor to linearize the pixel values before the step of normalizing the pixel 
values. 

In still another implementation of the method, the exposure value 
25 corresponding to each of the light-sensing elements is alterable by applying a 

respective exposure control signal to the mask. The method further comprises the 
steps of computing from the pixel values provided by the image sensor exposure 
control signals for minimizing the number of saturated pixel values and blackened 
pixel values from the image sensor, and applying the computed exposure control 
30 signals to the mask, as well as providing exposure values corresponding to the applied 
exposure control signals for use in the normalizing step. 
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In a still further implementation, the method comprises the step of applying a 
smoothing filter to the exposure values corresponding to the applied exposure control 
signals to derive smoothed exposure values corresponding to the normalized pixel 
values. The smoothed exposure values are then reapplied to the normalized pixel 
5 values by multiplying each normalized pixel value with a corresponding smooth 
exposure value to derive monitor image pixel values representing an image having 
compressed dynamic range but with visual details of the image preserved. 

In accordance with still another aspect of the present invention, there is 
provided a method for high dynamic range imaging comprising the steps of exposing a 

1 0 frame of photographic film in a film camera to an image of a scene through a planar 
mask disposed between the shutter and the focal plane of the camera* The mask has a 
multiplicity of light transmitting cells each controlling the exposure of a respective 
region of the film. Each of the cells of the mask has an associated exposure value 
indicative of the transparency of the cell. The exposure values are stored in an 

1 5 exposure pattern memory. The exposed film is then processed to contain a masked 
image of the scene. The masked image contained on the film or a print thereof is 
scanned to obtain pixel values representative of the different regions of the masked 
image exposed on the film through the cells of the mask. The masked image, as 
represented by the pixel values, is aligned with the mask as represented by the 

20 exposure values such that each pixel value corresponds to an exposure value 

associated with the mask cell through which the region represented by the pixel value 
was exposed. The pixel values are then normalized using corresponding exposure 
values to derive normalized pixel values disposed at respective positions of a first 
array. 

25 In one implementation of the method, there is included the further step of 

interpolating the normalized pixel values to derive interpolated pixel values at 
respective positions of a second array overlapping the first array. In another 
implementation of the method, there is further included the step of calibrating the 
pixel values according to a combined response function of the film and of the image 

30 sensor used to scan the masked image contained on the film. 
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BRIEF DESCRIPTION OF DRAWINGS 

For a more complete understanding of the nature and benefits of the present 
invention, reference should be made to the following detailed description taken in 
conjunction with the accompanying drawings, in which: 

Fig. 1 is a functional diagram of an imaging system according to a first 
exemplary embodiment of the present invention; 

Fig. 2 is a block diagram of the hardware components of a video camera 
implementation of the imaging system of Fig. 1; 

Fig. 3 is a data flow diagram of the image processor of the system of Fig. 1 ; 

Fig. 4 A is a plan view of the patterned mask used in the system of Fig. 1; 

Fig. 4B depicts a group of four neighboring cells that are repetitively disposed 
to form the mask pattern of Fig. 4 A; 

Fig. 5A depicts the imaging optics, the mask and image sensor of the system of 
Fig. 1 and objects in a scene whose image is being captured; 

Fig. 5B depicts the image of the objects of the scene as viewed through the 
patterned mask of Fig. 4A; 

Fig. 6 are diagrams illustrating calibration and normalization as carried out by 
the system of Fig. 1 on four neighboring pixel values as detected through four 
neighboring cells of the mask; 

Fig. 7 are diagrams illustrating calibration and normalization as carried out by 
the system of Fig. 1 on four neighboring pixel values as detected through four 
neighboring cells of the mask, where one of the pixel values is a blackened pixel 
value; 

Fig. 8 are diagrams illustrating calibration and normalization as carried out by 
the system of Fig. 1 on four neighboring pixel values as detected through four 
neighboring cells of the mask, where two of the pixel values are saturated pixel 
values; 

Fig. 9 are diagrams illustrating interpolation as carried out by the system of 
Fig. 1 , in particular the application of a 2 * 2 interpolation filter to every 2x2 pixel 
local area of the masked image to derive interpolated pixel values; 
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Fig. 1 OA is a diagram depicting the original image grid having light-sensing 
elements of the image sensor disposed at respective intersections thereof 
superimposed on an interpolation grid; 

Fig. 10B illustrates an interpolation filter applied to each different two pixel by 
two pixel region of the original image grid to derive interpolated pixels at respective 
intersections of the interpolation grid; 

Fig. 1 1 is a flow diagram of the operation of the image processor of the system 
of Fig. 1; 

Fig. 1 1 A is a flow diagram illustrating in greater detail the calibration step 208 
of the flow diagram of Fig, 1 1. 

Fig. 12 is a functional block diagram of an imaging system according to a 
second exemplary embodiment of the present invention; 

Fig. 13 is a block diagram depicting the hardware components of a video 
camera implementation of the imaging system of Fig. 12; 

Fig. 14 is a diagram depicting the arrangement of the optical components, the 
mask, a light diffuser and the light-sensing array of the system of Fig. 12; 

Fig. 15 is a diagram illustrating the correspondence of mask cells and the pixel 
positions of the image sensor of the system of Fig. 12; 

Fig. 16 are diagrams illustrating the operation of the mask controller of the 
system of Fig. 12 in adaptively controlling the exposure values of the mask; 

Fig. 17 is a flow diagram of the operation of the mask controller of the system 
of Fig. 12; 

Fig. 18 is a data flow diagram of the image processor of the system of Fig. 12; 

Fig. 19 are diagrams illustrating the smoothing operation of the exposure 
values and the reapplication of the smoothed exposure values to the captured high 
dynamic range image pixel values in the system of Fig. 12; 

Fig. 20 is a flow diagram of the exposure value smoothing operation carried 
out by the system of Fig. 12; 

Fig. 21 is a diagram representing an example of a smoothing filter useable in 
the smoothing operation of Fig. 20; 
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Fig. 22 is a functional diagram of an imaging system according to a third 
exemplary embodiment of the present invention; 

Fig. 23 is a diagram depicting the detailed structure of the aligner of the 
imaging system of Fig. 22; and 

Fig. 24 is a flow diagram of the operation of the aligner of the imaging system 
of Fig. 22. 

DETAILED DESCRIPTION 
Referring now to Fig. 1, there is shown a functional diagram of an imaging 
system 1 00 according to a first exemplary embodiment of the present invention. The 
system 100 includes a mask 101 through which incident light 2 from a scene (not 
shown) passes before impinging on an image sensor 3. The image sensor 3 senses an 
image of the scene as transmitted by the mask 101 and provides corresponding pixel 
values to an image processor 4, which is coupled to a response function memory 6 and 
an exposure pattern memory 5 and which provides high dynamic range image data to 
an output image memory 7. The mask 101 is a plate of transparent material which has 
neutral characteristics with respect to the wavelengths of the incident light and has a 
fixed spatial pattern of transparency in the region through which the incident light 2 
passes. The term "light* * as used in the specification and claims refers to 
electromagnetic radiation spanning the spectrum from the millimeter wave region to 
the gamma ray region. The image sensor 3 preferably comprises a charge coupled 
device (CCD) image detector having an array of light-sensing elements (not shown), 
but may be any other type of image sensor including photographic film. In general, 
the image sensor 3 will have a low dynamic range relative to the range of incident 
light 2 emanating from the scene. The response function data stored in memory 6 is 
the input/output characteristics of the image sensor 3, which has been previously 
measured and stored in the memory 6 as a look-up table, in parametric form or other 
appropriate data form. The exposure values stored in memory 5 are indicative of the 
transparency of the mask at each pixel position such that there is a corresponding 
exposure value for each pixel value provided by the image sensor 3. The exposure 
values were previously measured and stored in the exposure pattern memory 5 as raw 
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or coded data. The image processor 4 derives high dynamic range image data from 
the pixel values provided by the image sensor 3, the response function data stored in 
memory 6 and the exposure values stored in memory 5. The high dynamic range 
image data, which is stored in the output image memory 7, has radiance values 
proportional to corresponding radiance values of the scene. 

Turning now to Fig. 2, there are shown the hardware components of a video 
camera implementation 200 of the system of Fig. 1 . The camera of Fig. 2 has an 
image sensing part 401 , which includes a lens arrangement 402 for focusing an image 
of a scene onto the photosensitive surface of a CCD light-sensing array 165, an 
aperture 403 for controlling the overall exposure of the image, amask 101 positioned 
in close proximity to the CCD light-sensing array 1 65 and a preamplifier 404 for 
adjusting the CCD output and for reducing noise. The preamplifier 404 includes 
several analog circuits which are usually used in a video camera, such as a correlated 
double circuit, an automatic gain control circuit, a gamma correction circuit and a 
knee circuit. The camera implementation 200 of Fig. 2 also includes an image 
processing part 405 having an analog-to-digital converter 406 for digitizing the signals 
from the image sensing part 401 corresponding to the captured images. The digitized 
images are stored in a frame memory 407 which corresponds to the output image 
memory 7 in Fig. 1 . Computation of the high dynamic range image data is carried out 
in a processing unit 411, which has access to the data in the frame memory 407, a 
ROM 409 and a RAM 410. The ROM 409 is used to store the exposure values and 
the response function data, as well as the instructions for processing unit 411. The 
computed high dynamic range image in the frame memory 407 is converted to an 
analog signal by a digital-to-analog converter 408, which provides the analog signal 
to a video output part 412. The video output part 412, which is only needed in a video 
camera implementation and has no counterpart shown in Fig. 1 , includes a video 
encoder 413 which encodes the converted analog image to a video signal. The video 
signal is then passed through an output unit 414 which provides a video output 415. 

Fig. 3 shows a diagram 3000 of the data flow of the image processor 4 of Fig. 
1 . The image processor comprises calibrator 9, a normalizer 1 0 and an interpolator 
1 1 The calibrator 9 maps image data 8 captured by the image sensor 3 into a linear 
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response image by using the response function data stored in the response function 
memory 6. Then the nonnalizer converts the linear response image into an image 
which is proportional to the scene radiance. Finally, the interpolater 1 1 interpolates 
pixel values that cannot be computed because of saturation or blackening, and then 
5 provides the corrected image data to the output image memory 7. Interpolation may 
be used even in the absence of saturated or blackened pixels as a way of reducing 
noice. Fig. 4A shows an example of a mask pattern for the mask 101 used in the 
system of Fig. 1. In this mask example, the mask 101 is a two-dimensional array of 
cells with different transparencies (attenuations). The mask 101 consists of 

10 repetitively disposed identical groups of four cells having different transparencies. 
Fig. 4B shows the group of four cells 102, 103, 104 and 105, which is repetitively . 
disposed to form the pattern of the mask 101 of Fig. 4A. The group has a brightest 
cell 102, a bright cell 103, a dark cell 104 and a darkest cell 105 with the brightest cell 
102 being the most transparent and the darkest cell 105 being the least transparent of 

15 the four cells. While in the example of Figs. 4 A and 4B, a group of four cells are 
repetitively disposed to form the exposure pattern of the mask 101, groups having 
fewer or more cells having differing transparencies may be similarly repeated to form 
the same or a different exposure pattern. The exposure pattern of the mask need not 
be periodic and may even be random in distribution of transparencies. Moreover, the 

20 mask 101 may be constructed from a material that has a non-linear optical response. 
Such a material would have a transmittance (or attenuation) that varies in a non-linear 
manner with the intensity of the incident light. Such a mask would provide a detailed 
preserving optical filter that enhances the effective dynamic range of the imaging 
system while preserving visual details. 

25 Where a solid-state image sensing device such as a CCD or CMOS light- 

sensing array is used, the mask pattern can be fabricated on the same substrate as the 
image sensing device using photolithography and thin film deposition, or the mask 
pattern may be formed by photolithography and etching of the device itself. In the 
case of a film camera, the mask may be positioned between the shutter and the plane 

30 of the film or be attached to the film itself. In lieu of the mask, the film itself may 
have a spatially varying light sensitivity. The checkered mask 101 lets every local 



WO 00/79784 PCT/USOO/14515 

14 

area have four different exposures. Thus every local area is captured using the large 
dynamic range produced by the four different exposures. The exposure values of the 
cells in the checkered mask 101 are stored in the exposure value memory 5 to be used 
by the image processor 4. 

Figs. 5 A and 5B illustrate the capture of an image of a scene with the 
checkered patterned mask 101 . As shown in Fig. 5 A, a masked plate 1 01 is placed in 
front of an image sensing device 165, and the mask plate 101 is aligned so that each 
mask cell is positioned in front of a respective pixel (i.e., a light-sensing element) of 
the image sensing device 165. Incident light from the scene passes through the 
imaging optics 147, then focuses onto the image plane of the image sensor 3, but the 
light intensity which is attenuated by the masked plate 101 is recorded at each pixel 
position. Also shown in Fig. 5 A is a scene having a bright object 106 and a dark 
object 107. Thus an image like the one shown in Fig. 5B is obtained when the scene 
is captured by the image sensing device 165 through the masked plate 101. It should 
be noted that even if pixels of brighter exposure are saturated in an area of the bright 
object 106, pixels of darker exposure in the same area will still have values within the 
dynamic range of the image sensor 3. Similarly, even if pixels of darker exposure are 
too dark in an area of the dark object 107, pixels of brighter exposure in the same area 
will still have values within the dynamic range of the image sensor 3. 

Fig. 6 illustrates an example of the processing that is carried out by the 
calibrator 9 and the normalizer 10 of the image processor 4. In this example, a local 
area 1 12 of four neighboring pixels is taken from a region 1 1 1 in the background of 
the scene. Since the background of the scene is of constant brightness, the radiance 
values of the four pixels 1 12 are the same. However, the measured values of the four 
pixels differ because of non-linearities in the response function of the image sensor 3 
and the differences in the exposure values corresponding to the four pixel locations as 
a result of the transparency pattern of the mask. The diagram at the center of Fig. 7 
shows the process of converting each of the measured pixel values to a pixel value 
that is linear with respect to the scene radiance, and normalizing of the linearized 
pixel values. First, the calibration means 9 applies the response function of the image 
sensor 3 represented by the solid line response curve 1 13 to each of the measured 
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pixel values to remove the non-linearity of the image sensor 3, as represented by the 
dotted lines 1 14. Each of the calibrated pixel values is then divided by its 
corresponding exposure value, as represented by the dotted lines 115. The result is 
represented by the diagram 116, which shows four calibrated and normalized pixel 
values in neighboring pixel locations having the same radiance. While normalization 
as used in the example of Fig. 6 involves dividing a pixel value by its corresponding 
exposure value, it is contemplated that normalization encompasses any mapping of 
pixel values by a function of the exposure values. 

Fig. 7 illustrates another example of processing by the calibration means 9 and 
the nonnalizer 10 of the image processor 4, but this time the processing is on four 
neighboringpixellocationsll8takentomadarkregionll7ofthescene. Inthe 
example of Fig. 7, three of the pixel locations have pixel values in the dynamic range 
of the image sensor 3 but one of the pixel locations has a blackened pixel value. 
Calibration and normalization of the four pixel values is shown in the diagram at the 
center of Fig. 1. In the pixel value axis of the diagram, the blackened pixel threshold 
and the saturated pixel threshold are indicated by T b and T s , respectively. As shown in 
the diagram, the calibration and normalization process cannot be correctly applied to 
the blackened pixel value, as indicated by the broken line 119. Thus, the result of the 
calibration and normalization is represented by the diagram 120, which shows three 
calibrated and noimalized pixel values having the same radiance and one blackened 
pixel value (marked with the letter "b") as to which calibration and normalization 
cannot be correctly applied. 

Turning now to Fig. 8, there is shown yet another example of processing by 
the calibrator 9 and the nonnalizer 10 of the image processor 4, but this time the 
processing is on four neighboring pixel locations 122 taken fiom a bright region 121 
of the scene. Two of the four pixel locations 122 have pixel values in the dynamic 
range bf the image sensor 3 while the other two have saturated pixel values. 
Calibration and normalization of the four pixel values are illustrated by the diagram at 
the center of Fig. 8. Once again the blackened pixel threshold and the saturated pixel 
threshold are indicated by T b and T„ respectively, on the pixel value axis of the 
diagram. The two saturated pixel values cannot be correctly calibrated or normalized 
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as indicated by the broken line 123. The result of the processing is represented by the 
diagram 124, which shows calibrated and normalized pixel values of the same 
radiance at two pixel locations and saturated pixel values (marked with the letter "s") 
at the other two pixel locations. 

In accordance with the present exemplary embodiment, saturated or blackened 
pixel values are marked by the calibrating means 9 so that the interpolator 1 1 can treat 
such exceptions accordingly. 

Fig. 9 illustrates the interpolation process as carried out by the interpolator 1 1 
of the image processor 4. At the upper right hand portion of the figure, there is shown 
a 10 x 10 group of pixels 126 representing the region 125 of the scene, which 
encompasses a boundary of dark and bright portions of the scene. The pixel values at 
the respective pixel locations 126 have undergone the calibration and normalization 
process described in connection with Figs. 6, 7 and 8 and all saturated pixel are 
marked with a black v and all blackened pixel values are marked with a white "x." 
The interpolator applies a two pixel by two pixel interpolation filter to every different 
two pixel by two pixel region of the image pixel amy. The interpolation filter 
computes an interpolated pixel value at the center of each two pixel by two pixel 
region by taking the average of non-saturated and non-blackened pixel values in the 
region. Since every different two pixel by two pixel region is exposed through four 
different transparencies of the mask, there will generally be at least one non-saturated 
and non-blackened pixel value in each region. At the position of the interpolation 
filter 127 shown in the Fig. 9, the two pixel by two pixel region encompassed by the 
interpolation filter includes two saturated pixel values, which are not taken into 
account by the interpolation process. As a result, an interpolated pixel 129 is obtained 
at the center of the two pixel by two pixel region having a value which is the average 
of two non-saturated and non-blackened pixel values in the region. Alternatively, 
saturated and blackened pixels may be assigned respective predefined values and are 
included in the interpolation computation. 

Advantageously, each pixel value is weighted in accordance its contribution in 
increasing the quality of the interpolated result. Because the four neighboring pixels 
are normalized by different exposure values, the signal-to-noise ratio (SNR) of each of 
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the four pixels differs from each other. By weighting each pixel value with the SNR, 
the interpolation is made more immune to noise. Given the response function f(M) of 
pixel value M, the SNR of the pixel value can be computed as SNR (M) = f(M)/f(M), 
where f (M) is the first derivative of f(M). 
5 The interpolation process earned by the interpolator 1 1 of the image sensor 4 

is further illustrated in Figs. 10A and 10B. Referring to Fig. 10A, there is shown an 
original image grid 150 defined by spaced intersecting perpendicular lines extending 
in a vertical direction (the y direction) and in a horizontal direction (the x direction). 
The image sensor 3 comprises an array of light-sensing elements 151 disposed at 

1 0 respective intersections (pixel position) of the original image grid 1 50. The image 
sensor 3 providing a respective pixel value corresponding to each of the light-sensing 
elements when exposed to incident light Overlapping the original image grid 1 50 is 
an interpolation grid 152 also defined by spaced intersecting perpendicular lines 
extending in the vertical direction and in the horizontal direction. Referring now to 

1 5 Fig. 1 0B, the interpolation filter 1 53 encompasses each different four intersection 
regions of the original image grid 150 to derive interpolated pixel values 154 at 
respective intersections of the interpolation grid. In the present exemplary 
embodiment, the interpolation grid is identical to the original image grid and is 
displaced from one another by a one-half grid position in the horizontal direction and 

20 a one-half grid position in the vertical position. It will be understood by those skilled 
in the art that the interpolation grid need not be identical to the original image grid and 
that interpolation filters other than a two pixel by two pixel averaging filter may be 
used, such as a bi-linear interpolation filter, a bi-cubic interpolation filter, a B-spline 
interpolation filter, a Gaussian interpolation filter or any other interpolation filter 

25 known to those skilled in the art. Moreover, it is noted that the disposition of the 
light-sensing elements need not be at intersections of a grid but may be any form of 
array. 

Referring now to Fig. 1 1, there is shown a flow diagram 300 representing the 
processing which is carried out by the image processor 4. First, the process of loop 
30 201 is repeated for each pixel location, where x and yaie the x,y coordinates of the 
pixel location and xSize and ySize are the x dimension and the y dimension of the 
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pixel array. Loop 201 includes steps 202, 203, 204, 205, 206, 207, 208 and 209. In 
step 202, pixel value M(xy) at pixel location (xy) is evaluated. If M(xy) is smaller 
than noiseLevel, which is a predefined small pixel value, then step 203 is carried out 
If M(xy)is not smaller than noiseLevel, then step 205 is performed. In step 203, the 
5 exposure value E (xy) corresponding to the pixel location is read from exposure 

pattern memory 5. If E(xj>) is the highest exposure value, then step 205 is performed. 
Otherwise, step 204 is carried out. In step 204, the pixel is marked as having a 
"blackened" pixel value and the current iteration of the loop terminates. The marking 
is done by writing a special value into the pixel location in temporary memory. In 

1 0 step 205, pixel value M (xy) is evaluated. If M (xy) is larger than saturationLevel, 
which is a predefined very large pixel value, then step 206 is carried out. If Mfoy), is 
not larger than saturationLevel, then steps 208 and 209 are carried out. In step 206, 
the exposure value E (xy) is read from memory 5. If E(xj>) is the lowest exposure 
value, then steps 208 and 209 are performed. Otherwise step 207 is carried out. In 

15 step 207, the pixel is marked as a "saturated" pixel value in temporary memory, and 
the current iteration of loop 201 is terminated. In step 208, M(xy) is calibrated in a 
manner described in connection with Figs. 6, 7 and 8. In step 209, M(xy) is 
normalized as described in connection with Figs. 6, 7 and 8, and the current iteration 
of loop 201 is terminated. 

20 Turning to Fig. 1 1 A, there is shown a flow diagram of the details of the 

calibration step 208 of the flow diagram of Fig. 1 1, as carried out by the calibrator 9. 
In step 31 1, the pixel value M(xj/) at each pixel position (xy) is provided as an input 
M. In step 312, the response function f is applied to M to obtain m. As described 
above, the response function can be in the form of a look-up table, a parametric form 

25 or other appropriate data form. Assuming that the response function data is in the 
form of look-up table with 256 levels, M is normalized and digitized into an index 
value indicating one of the 256 levels. The look-up table is then used to find f(M) 
with the index value corresponding to M. In step 3 13, M is provided as an output, 
which in the flow diagram of Fig. 1 1 is the pixel value M(xj>) provided to step 209. 

30 Returning to Fig. 1 1 , after loop 201 is completed for all pixel positions, loop 

210 is processed. Loop 210 is repeated for each pixel position from location (1,1) to 
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location (xSize - 1, ySize - 1). In loop 210, step 21 1, loop 212, steps 213, 214 and 
21 5 are carried out. In step 21 1 , Lfay)and N are both initialized to zero. Here, 
L (revindicates the radiance value at position (xj>) in the output image memory 7 and 
N indicates the number of pixels processed. Then the process of loop 212 is repeated 
with 07). Both , i mdj vary from 0 to 1. In loop 212, steps 213 and 214 are carried 
out. In step 213, the marking of pixel (x+r>+/)*s determined . If the pixel is not 
marked, then step 214 is performed. Otherwise, the current iteration of the loop 
terminates. In step 214, weight value W is calculated as the SNR of pixel value 
M(x+iy+f). Then pixel value M(x+iy+j) is multiplied by W and the product is 
added to L fay), and W is added to N. After loop 212 is finished, step 2 15 is carried 
out. In step 215, Lfay) is divided by N, after the which the loop terminates. After the 
loop 210 is completed for all pixel locations, the processing by the image processor 4 
is completed. 

Referring again to Fig. 1, the imaging system according to the first exemplary 
embodiment of the present invention may be modified by eliminating the mask 101 
and using an image sensor 3 comprising an array of light-sensing elements (pixels) 
having a spatially vaiying pattern of photosensitivities. The photosensitivity pattern 
of the light-sensing elements are stored as respective sensitivity values in memory 5 in 
place of the exposure pattern. A respective response function for each pixel is stored 
in memory 6 in place of the single response function for the image sensor. Turning 
again to Fig. 3, the calibrator 9 receives pixel values of the captured image 8 and uses 
the respective response functions in memory 6 to linearize the pixel values. The 
linearized pixel values are then normalized by the normalizer 10 using the 
photosensitivity values stored in memory 5. The normalized pixel values are then 
interpolated in the manner described to derive the high dynamic range image pixel 
values. 

Turning now to Fig. 12, there is shown a functional block diagram 400 of an 
imaging system according to a second exemplary embodiment of the present 
invention. The system according to the second exemplary embodiment comprises a 
variable mask 12, a mask controller 13, an image sensor 3, an exposure pattern 
memory 5, a response function memory 6, an output image memory 7, an image 
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processor 14, and a monitor image memory 15. Like the system of Fig. 1, incident 
light from a scene passes through a mask 12 having a spatially varying transparency 
pattern before impinging on the light sensitive surface of an image sensor 3 which 
provides corresponding pixel values to an image processor 14. The image processor 
14 also receives response function data from the response function memory 6 and 
exposure values from an exposure pattern memory 5 and provides high dynamic range 
image data to an output image memory 7. However, unlike the system of Fig. 1, the 
system 400 of Fig. 12 employs a mask 12 having a transparency pattern that may be 
varied by the application of exposure control signals thereto, and a mask controller 1 3 
that responds to the pixel values provided by the image sensor 3 for generating the 
exposure control signals applied to the mask 12 and corresponding exposure values 
provided to the exposure pattern memory 5. In addition, the image processor 14 
provides monitor image data to be stored in the monitor image memory 1 5. The 
monitor image data have a compressed dynamic range with image detail preservation 
for display by a low dynamic range display device, such as a television, a video 
monitor or a computer display. The mask controller 13 receives the output of the 
image sensor 3, and adjusts the spatially varying transparency of the mask 12 such that 
saturated and blackened pixel values are minimized. 

Turning now to Fig. 13, there is shown a block diagram of 500 of the hardware 
components of a video camera implementation of the system of Fig. 12. The video 
camera implementation comprises an optical attachment 161 having a first lens 
component 162 for focusing a scene onto the variable mask 12 and a second lens 
component 163 for focusing the image of the scene after passing through the mask 12 
and a light diffuser 131 onto the light sensing surface of a CCD light-sensing array 
165. The optical attachment 161 also includes a liquid crystal mask controller 183 for 
applying mask control signals to the liquid crystal mask 12 for varying its 
transparency pattern. The block diagram 500 also includes an image sensing part 164 
having a CCD light-sensing array 165 and a preamplifier 166 for adjusting the 
amplitude of the CCD output and for reducing noise. The preamplifier 1 66 includes 
several analog circuits which are usually implemented in a video camera, such as a 
correlated double circuit (a noise reduction circuit), an automatic gain control circuit, 
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a gamma correction circuit and a knee circuit The output of the preamplifier 166 is 
provided to an analog-to-digital converter 169 in the image processing part 167. The 
analog-to-digital converter 169 digitizes the captured images. The actual computing 
of the image processor 14 of Fig. 12 is carried out by the processing unit 168, which 
has access to a RAM 170,aROM 171, a Frame Memory 1 172, a Frame Memory 2 
178, and a Frame Memory 3 1 74. In the video camera implementation 500 of Fig. 1 3, 
a liquid crystal mask controller 183 provides the controlling voltage to the liquid 
crystal mask 12. The controlling voltages applied to the liquid crystal mask are 
computed by the Processing Unit 168. The ROM 171 provides storage for 
instructions executed by the Processing Unit 168, as well as control parameters. The 
RAM 170 provides temporary data storage during processing. Frame Memory 1 172 
corresponds to the output image memory 7 of Fig. 12, Frame Memory 2 173 
corresponds to the Exposure Pattern Memory 5 in Fig. 12, and Frame Memory 3 174 
corresponds to the Monitor Image Memory 15 of Fig. 12. The high dynamic range 
image data stored in Frame Memory 1 1 72 is converted to an analog signal by a 
digital-to-analog converter 175 before being provided to Video Output Part 1 177, 
where the analog signals undergo processing by a Video Signal Processor 178, the 
output of which is provided to Output Unit 179, which provides a video output 181 of 
the high dynamic range image. The monitor image data in Frame Memory 3 174 is 
converted to an analog signal by digital-to-analog converter 176. The analog signal 
for the monitor image is provided to Video Output Part 2 180 where in undergoes 
processing by Video Signal Processor 1 81, the output of which is provided to Output 
Unit 1 82, which provides the video output 182 of the monitor image. It is noted that 
while Video Output Part 1 177 and Video Output Part 2 180 are part of the video 
camera implementation, these components have no counterparts in the basic imaging 
system diagram of Fig. 12. 

Turning now to Fig. 14, there is illustrated an exemplary structure of the image 
capture components 600 which may be used in the first and second exemplary 
embodiments of the present invention. A fixed pattern or liquid crystal mask 130 and 
a light difruser 13 1 are positioned parallel to the image plane of a CCD light-sensing 
array 165, which is embedded in a camera body 134. The mask 130 and a light 
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diffiiser 131 are disposed between two optical components 132 and 133. The incident 
light from a scene is focused onto the mask 130 by the first optical component 132. 
Light attenuated by the mask 1 30 and passed through the light diffuser 1 3 1 is focused 
onto the image plane of the CCD light- sensing array 165 by the second optical 
component 133. In this manner, the light measured by the CCD light-sensing array 
165 is in focus with respect to the mask 130 and the scene. It is noted that the light 
impinging on a pixel position R of the CCD image sensing device 165 comes only 
from a small area Q of the mask 1 30 through which the light has passed. The light 
difiuser 131 positioned directly behind the mask 130 removes directionality of 
focused light Alternatively, the light diffuser 131 may be positioned directly in front 
of the mask 1 30 or two separated light diffusers may be positioned one directly in 
front and one directly behind the mask 130. The small area Q in turn receives light 
only from a single point P of the scene that is being imaged. Although a liquid crystal 
array is the preferred variable exposure mask for the second exemplary embodiment 
of the invention, other devices, such as an array of electro-optic light modulators, 
where the exposure pattern can be controlled by the application of appropriate signals 
may also be used. 

As mentioned above, it is preferred that the variable mask 12 be a two- 
dimensional array of liquid crystal cells having transparencies that are individually 
controllable by the application of respective exposure control voltage signals. As 
shown in Fig. 15, each cell of the variable mask 12 controls the transmission of light 
to a respective one or more of the pixels of the light-sensing element array 165 of the 
image sensor 3, where each of the light-sensing elements of the array 165 corresponds 
to a respective pixel. For example, the 4 x 4 pixel area which is indicated by the 
dotted square 183 in the light-sensing element array 165 receives only light that has 
passed through the mask cell 184 of the variable mask 12. The exposure control 
signals that control the transparency (attenuation) of each cell of the variable mask 12 
are provided by a mask controller 13 which receives the output (pixel values) of the 
image sensor 3 and computes the exposure control signals for the cells of the mask 
according to the received pixel values. The exposure control signal for each cell is 
computed so as to (1) minimize the number of saturated or blackened pixels within 
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the corresponding area 183 of the light-sensing element airay 165, (2) maximize the 
signal-to-noise ratio of the average pixel value provided by the image sensor 3, and (3) 
maximize the spatial smoothness of the exposure values of the mask cell array 12. 

Turning now to Fig. 16, there is illustrated an example of the evolution of the 
exposure pattern of the variable mask 12 under the control of the mask controller 13. 
In this figure, time shifts from left to right. The illustrations 1 37 at the top row of the 
figure represent the radiance from the scene, which in the present example remains 
constant over time. The illustrations 138, 139 and 140 in the center row of the figure 
represent the temporal change of the exposure pattern of the mask 12. The 
illustrations 141, 142 and 143 in the bottom row of the figure depict the temporal 
change of the captured image as represented by the output of the image sensor 3. In 
the initial state (left column), the exposure pattern 138 starts out with the same 
transparency (attenuation) for all of the cells of the variable mask 12. Thus, the 
radiance of the scene 137, which has a very bright object 1 85 and a very dark object 
186. are equally attenuated to produce the initial captured (un-normalized) image 141. 
However, since the dynamic range of the image sensor 3 is low relative to the range of 
radiance of the scene, the initial captured image 141 does not include all of the visual 
details of the radiance scene 137. In the initial captured image 141, the dark object 
1 87 is almost fully blackened and the bright object 1 88 is almost fully saturated. After 
the mask controller 13 has adjusted the exposure values of every cell of the variable 
mask 12 to satisfy the three criteria mentioned above, the second exposure pattern 139 
results from the mask controller 13 applying new exposure control signals to the 
variable mask 12. In the exposure pattern 139, the exposure values of the bright 
object areas are decreased, the exposure values at the dark object areas are increased 
and the exposure values at the background areas are slightly increased. The image 
1 42 captured using the exposure pattern 1 39 has no saturated or blackened pixels, and 
hence the details of the image are enhanced. In a like manner, the mask controller 13 
continues to adjust the exposure values to obtain exposure pattern 140 which has 
improved smoothness. The image 143 obtained with the exposure pattern 140 shows 
improved image detail as a result of the improved the smoothness of the exposure 
pattern. It is noted that the process of adapting the exposure pattern by the exposure 
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controller 13 is particularly suited to temporally varying scenes, such as scenes with 
moving objects. 

Referring now to Fig. 17, there is shown a flow diagram 700 of the processing 
carried out by the mask controller 1 3. The flow diagram 700 represents the process 
for computing an exposure control signal for one cell of the variable mask 12, and 
must therefore be repeated for each cell of the mask. In step 216, S, and S 2 are both 
initialized to zero. Here, S, indicates the exposure control value for satisfying the first 
criterion of minimizing the number of saturated or blackened pixel values within the 
area of the light-sensing element array that receives light from the mask cell, and S 2 
indicates the exposure control value that satisfies the second criterion of maximizing 
the signal-to-noise ratio of the average pixel value provided by the image sensor 3. 
The process of loop 2 1 7 is repeated for each pixel within the area of the light-sensing 
element array 165 that receives light which has passed through the cell of the variable 
mask 12. Here, Q is the number of pixels within that area. The loop 217 includes 
steps 218, 219, 220, 221 and 222. In step 218, the pixel value M, is evaluated. If M„ 
is smaller than noiseLevel, which is a predefined very small pixel value, then step 219 
is carried out. Otherwise, step 222 is performed. In step 219, S, is incremented and 
the current iteration of loop 217 terminates. In step 220, pixel value M q is evaluated. 
If M q is larger than saturationLevel, which is a predefined very large pixel value, then 
step 221 is carried out. Otherwise, step 222 is performed. In step 221, S, is 
decremented, and the current iteration of the loop terminates. In step 222, S 2 is 
updated by adding the quantity (M^, - fcy, where M^, is a predefined pixel value 
which has the best signal-to-noise ratio for the image sensor 3 being used. After loop 
217 is finished, steps 223 and 224 are carried out. In step 223, S 3 is computed by 
using the equation shown in block 223. In this equation, g k are weight values of a low 
pass kernel, E,, are exposure values of the neighbor cells, and E is the exposure value 
of the currently processed cell. This computation provides a type of low pass filtering, 
which is commonly called "unsharp masking." In step 224, the exposure value E of 
the currently processed cell is updated by a weighted summation of S„ S 2 and S 3> 
where the weights w„ w 2 and w 3 are predefined balancing factors experimentally 
determined to optimize the image quality. After step 224 is finished, the processing 
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for the currently processed cell is completed. 

Turning now to Fig. 18, there is shown a data flow diagram 800 of the image 
processor 14 of the system of Fig. 12. The image processor 14 comprises a calibrator 
9, a nonnalizer 10, a smoother 16 and an exposure reapplicator 17. As with the image 
processor 4 of the first exemplary embodiment of the present invention illustrated in 
Fig. 1 , the calibrator 9 calibrates the pixel values of an image captured by image 
sensor 3 into a linear response image by using the response function data stored in the 
response function memory 6. Then the nonnalizer 10 converts the linear response 
image into an image which is proportional to the scene radiance. It is noted that the 
image processor 14 of the second embodiment of the present invention does not 
require interpolation of the output of the nonnalizer 10 because the exposure of the 
variable mask 12 is adjusted by the mask controller 1 3 to minimize saturated and 
blackened pixel values. 

Furthermore, the image processor 14 produces monitor image pixel values to 
be written into the monitor image memory 15. To produce the monitor image pixel 
values, smoother 16 smooths the exposure values from the exposure pattern memory 
5. Then the exposure reapplicator 17 applies the smoothed exposure values to the 
pixel values of the high dynamic range image as produced by the nonnalizer 10. Fig. 
20 illustrates the process for deriving the monitor image pixel values. The left hand 
column of Fig. 19 is the same as the right hand column of Fig. 16. Because after 
exposure control the captured image 143 still has slight discontinuities caused by the 
boundaries of the mask cells, it is not ideally suited for display purposes. Therefore, 
the smoother 16 smooths the exposure pattern 140 to derive the smoothed exposure 
pattern 145, which has no visible discontinuities. The monitor image 146 is then 
obtained by reapplying the smoothed exposure pattern 145 to the high dynamic range 
image 144, which has already been obtained from exposure pattern 140 and the 
captured (un-normalized) image 143. The monitor image 146 is better suited for 
display purposes than the captured image 143 or the high dynamic range image 144 
because the monitor image 1 46 has no discontinuities and has a dynamic range similar 
to that of the image sensor 3. The monitor image may also serve as video output for 
subsequent processing 
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Referring to Fig. 20, there is shown a flow diagram 900 of the processing 
carried out by the smoother 16 and the exposure reapplicator 1 7. With respect to 
smoothing, loop 301 is carried out N times, where N is a predetermined number, 
usually in the range tram 1 to 5. In loop 301, step 302, loop 303, step 304 and step 
305 are carried out. In step 302 exposure pattern data E in the exposure pattern 
memory 5 is copied to another memory space of the same size and is referred to as E r 
In the example of Fig. 20, the exposure partem has the same dimensions as the image 
so that each pixel of the image has the same coordinates as its corresponding exposure 
value. The loop 303 is repeated for each pixel position of E^, which is a temporary 
exposure pattern data storage of the same size as E and E,. In loop 303, step 304 is 
processed. In step 304, a smoothing filter g is applied to the current position (x,y) of 
E,. The filtering computation is expressed in the box of step 304. The smoothing 
filter g(u,v) may be that shown in Fig. 21 . Referring to Fig. 21, there is shown a 5 x 5 
(k=5) Gaussian blur filter. Each number in the grid 501 expresses the filter coefficient 
at that position of the filter. For example, g(3,3) = 0.16. Referring again to Fig. 20, 
the computed value in step 304 is stored in the same position (x,y) of E^,. After loop 
303 is finished, step 305 is carried out. In step 305, the smoothed exposure values in 
are copied to E s . After loop 301 has been repeated N times, loop 306 is carried 
out to implement exposure reapplication. The loop 306 is repeated for each pixel 
position of E,. In loop 306, step 307 is carried out. In step 307, the pixel value L(x,y) 
in the output image memory 7 and the exposure value E,(x,y) are multiplied and the 
product of the multiplication is stored in the same pixel position of the monitor image 
memory 1 5 as (x,y). When loop 306 is completed, processing for smoothing of 
the exposure pattern and for reapplication of the smoothed exposure of pattern to the 
high dynamic range image ends. 

Turning now to Fig. 22, there is shown a functional diagram of a system 1000 
for capturing high dynamic range images of a scene according to a third exemplary 
embodiment of the present invention. The system includes a photographic film 
camera 509 for exposing a masked image of a scene onto photographic film. As 
mentioned above, the masked image may be obtained by providing a mask 101 having 
a spatially varying transparency pattern, such as that shown in Fig. 4A, between the 
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shutter 504 of the camera and the plane of the film 505. Alternatively, the mask may 
be attached to each frame of film or the film emulsion may have a spatially varying 
pattern of exposure sensitivity. In each case, the exposure value for each region of the 
film has been previously determined and is stored in an exposure pattern memory 5. 
5 After being exposed with a masked image, the film is processed to produce either a 
transparency or a print containing the masked image. The film or print containing the 
masked image is then scanned by a scanner 507 which provides pixel values 
representative of the masked image. The pixel values are provided to an image 
processor 4 which may be identical to the image processor of the system of Fig. 1 and 

1 0 may have a data flow diagram identical to the one shown in Fig. 3. The combined 
response function of the film and the scanner are stored in a response function 
memory 6 to be used by the image processor 4 for calibration purposes. Instead of 
receiving exposure values directly from the exposure pattern memory 5, the system 
1000 of Fig. 22 includes an aligner which receives the output of the scanner 507 and 

15 the exposure values stored in exposure pattern memory 5, and provides alignment 
corrected exposure values to the image processor for normalization purposes. The 
image processor 4 provides a high dynamic range output image to an output image 
memory 7. In typical film cameras, the film is moved frame by frame with each frame 
being positioned behind the shutter 504. This movement of the film causes some 

20 misalignment between the currently exposed frame of the film 505 and the mask 101. 
The aligner 508 is used to correct for such misalignment. 

Turning now to Fig. 23, there is shown a data flow diagram 1 100 of the aligner 
508 in Fig. 22. The aligner 1 100 has three fast Fourier transform processors 512, 513 
and 514, a rotation detector 515, a shift detector 516, a rotator 517 and a shifter 518. 

25 The process carried out by the aligner 508 is represented by the flow diagram in Fig. 
24. Referring to Fig. 24, in step 519 the Fourier transform I(u,v) of the image I(xj;) is 
computed by FFT1 512. In step 520, the Fourier transform E(u,v) of the exposure 
pattern E(xj>) is computed by FFT2 513. In step 521, the positions of the peaks of the 
magnitude of I(u,v) are detected. In step 522, the positions of the peaks of the 

30 magnitude of E(u,v) are detected. In step 523, a 2 x 2 rotation matrix M is estimated 
by least square optimization with the error function which is shown in block 523. 
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Steps 521 , 522 and 523 are all carried out by the rotation detector 5 1 5 in Fig. 23. In 
step 524, a rotated exposure pattern (xy) is computed by rotating the exposure 
pattern E(xj>) with the rotation matrix M. Step 524 is carried out by the rotator 517 
of Fig. 23. In step 525, a shift vector (s,t) is estimated by searching for the shift vector 
which minimizes the error function shown in box 525. Finally, in step 526 the 
corrected exposure pattern E,^^ (xy) is computed by shifting E^^xy) by the 
shift vector (s,t). The corrected exposure pattern E^^ (xy) is provided to a 
corrected exposure pattern memory 5 19 for use by the image processor 4 for 
normalization purposes. 

While the present invention has been particularly described with reference to 
exemplary embodiments thereof, it will be understood by those skilled in the art that 
various modifications and alterations may be made without departing from the spirit 
and scope of the invention. Accordingly, the disclosed embodiments of the invention 
are considered merely illustrative, and the invention is limited in scope only as 
specified in the appended claims. 
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CLAIMS 

1 . A system for high dynamic range imaging comprising: 

an image sensor comprising an array of light-sensing elements, the 
image sensor sensing an image of a scene and providing corresponding pixel values 
representing light intensities impinging on respective ones of the light-sensing 
elements, the image sensor providing a saturated pixel value when the intensity of 
light impinging on a corresponding one of the light-sensing elements is greater than a 
first threshold level and providing a blackened pixel value when the intensity of light 
impinging on a corresponding one of the light-sensing elements is below a second 
threshold level, the image sensor having a low dynamic range relative to the range of 
light intensities of the image of the scene; 

a mask interposed between the scene and the array of light-sensing 
elements of the image sensor, the mask having a multiplicity of light-transmitting cells 
each controlling the exposure of a respective one or more of the light-sensing 
elements to light from the scene, each of the light-sensing elements having a 
corresponding exposure value indicative of the transparency of the cell through which 
light impinging on the light-sensing element passes; 

a first memoiy storing exposure values corresponding to the light- 
sensing elements; and 

an image processor coupled to the image sensor for receiving the pixel 
values provided thereby and coupled to the first memory for receiving the exposure 
values corresponding to the pixel values, the image processor comprising a normalizer 
for mapping the pixel values by a function of the exposure values to derive 
corresponding normalized pixel values. 

2. The system according to claim 1 , wherein the normalizer divides 
each of the pixel values with the exposure value corresponding to the light-sensing 
element receiving light intensity represented by the pixel value to derive 
corresponding normalized pixel values. 
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3. The system according to claim 1, wherein the exposure values 
corresponding to the light-sensing elements are fixed and the image processor further 
comprises an interpolator for interpolating the normalized pixel values to derive 
interpolated pixel values at respective positions of a second array overlapping the 
array of the light-sensing elements. 



4. The system according to claim 3, wherein the interpolator of the 
image processor interpolates the normalized pixel values, excluding normalized pixel 
values corresponding to saturated pixel values or blackened pixel values. 



5. The system according to claim 3, wherein the interpolator of the 
image processor interpolates the normalized pixel values multiplied by respective 
weighting values. 



6. The system according to claim 5, wherein the weighting values are 
respective signal-to-noise ratios of the corresponding pixel values. 

7. The system according to claim 3, wherein the interpolator of the 
image processor applies an interpolation filter to the normalized pixel values. 

8. The system according to claim 3, wherein the light-sensing elements 
of the array are disposed at respective intersections of a first grid defined by spaced 
intersecting perpendicular lines extending in a vertical direction and a horizontal 
direction, and the interpolator derives interpolated pixel values at respective 
intersections of a second grid defined by spaced intersecting perpendicular lines 
extending in the vertical direction and in the horizontal direction, the first and second 
grids overlapping one another. 



9. The system according to claim 8, wherein the interpolator of the 
image processor separately interpolates normalized pixel values in each different 
region of the first grid, each region overlapping adjacent regions and encompassing a 
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predetermined number of intersections of the first grid. 



1 0. The system according to claim 9, wherein the mask comprises 
repetitively disposed identical groups of cells having different transparencies. 

1 1 . The system according to claim 10, wherein each of the repetitively 
5 disposed identical groups of cells of the mask consists of four neighboring cells 

having different transparencies, each cell controlling the light exposure of a different 
one of the light-sensing elements, and the first and second grids have identical 
dimensions and are displaced from one another by one-half grid position in the 
horizontal direction and one-half grid position in the vertical direction. 

10 12. The system according to claim 9, wherein the interpolator of the 

image processor derives a respective interpolated pixel value in each different region 
of the first grid by taking the average of all normalized pixel values in the region 
which do not correspond to a saturated pixel value or a blackened pixel value. 

13. The system according to claim 1 further comprising a second 
1 5 memory storing a response function of the image sensor, wherein the image processor 
is coupled to the second memory for receiving the response function and further 
comprises a calibrator for converting the pixel values provided by the image sensor to 
linear response pixel values in accordance with the response function before the pixel 
values are normalized by the normalizer. 

20 14. The system according to claim 3 further comprising an output 

image memory coupled to the image processor for receiving and storing the 
interpolated pixel values. 



25 



15. The system according to claim 1 further comprising a first lens 
system located between the scene and the mask for projecting an image of the scene 
onto the mask and a second lens system for projecting the image of the scene as 
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transmitted through the mask onto the array of light-sensing elements. 

16. The system according to claim 15, further comprising a light 
diffuser adjacent to the mask for removing directionality of light focused by the first 
lens system. 

5 17. The system according to claim 1 , wherein the array of light sensing 

elements is a solid state device image sensing device. 

1 8. The system according to claim 17, wherein the solid state device is 
a charge-coupled device light sensing array. 

19. The system according to claim 18, wherein the solid state device is 
10 a CMOS light-sensing array. 

20. The system according to claim 17, wherein the mask is integrated 
with the light-sensing elements on the same substrate. 

21. The system according to claim 1, wherein the mask is made of a 
nonlinear optical material. 

1 5 22. A system for high dynamic range imaging comprising: 

an image sensor comprising an array of light-sensing elements, the 
image sensor sensing an image of a scene and providing corresponding pixel values 
representing light intensities impinging on respective ones of the light-sensing 
elements, the image sensor providing a saturated pixel value when the intensity of 

20 light impinging on a corresponding one of the light-sensing elements is greater than a 
first threshold level and providing a blackened pixel value when the intensity of light 
impinging on a corresponding one of the light-sensing elements is below a second 
threshold level, the image sensor having a low dynamic range relative to the range of 
intensities of the image of the scene; 
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a mask interposed between the scene and the array of light-sensing 
elements of the image sensor, the mask having a multiplicity of light-transmitting cells 
each controlling the exposure of a respective one or more of the light-sensing 
elements to the light from the scene, each of the light-sensing elements having a 
5 corresponding exposure value indicative of the transparency of the cells through 

which light impinging on the light-sensing element passes, the exposure value of each 
of the cells of the mask being alterable by applying a respective exposure control 
signal to the mask; 

a mask controller coupled to the image sensor for receiving the pixel 
1 0 values provided thereby and for computing, and applying to the mask, exposure 
control signals for minimizing the number of saturated pixel values and blackened 
pixel values from the image sensor, and providing exposure values corresponding to 
the applied exposure control signals; and 

an image processor coupled to the image sensor for receiving the pixel 
1 5 values provided thereby, the image sensor comprising a normalizer for mapping the 
pixel values by a function of the exposure values to derive corresponding normalized 
pixel values. 

23. The system according to claim 22, wherein the normalizer of the 
image processor divides each of the pixel values by the exposure value corresponding 

20 to the light-sensing element receiving light intensity represented by the pixel value to 
derive corresponding normalized pixel values. 

24. The system according to claim 22, further comprising an exposure 
pattern memory coupled to the mask controller for receiving and storing the exposure 
values, and wherein the image processor is coupled to the exposure pattern memory 

25 for receiving the exposure values corresponding to the pixel values from the image 
sensor. 



25. The system according to claim 22 further comprising an output 
image memory coupled to the normalizer of the image processor for receiving and 



WO 00/79784 

storing the normalized pixel values. 



34 



PCT/USO0/14515 



26. The system according to claim 24 further comprising: 

a smoother coupled to the exposure pattern memory for receiving 
exposure values stored therein and applying a smoothing filter to the exposure values 
to derive smoothed exposure values corresponding to the normalized pixel values; 

an exposure reapplicator coupled to the normalizer of the image 
processor for receiving the normalized pixel values and for multiplying each of the 
normalized pixel values with a corresponding smoothed exposure value to derive 
monitor image pixel values; and 

a monitor image generator coupled to the exposure reapplicator for 
providing the monitor image pixel values. 

27. The system according to claim 22, wherein the mask controller 
computes the exposure control signals to maximize the signal-to-noise ratio of the 
average pixel value provided by the image sensor 

28. The system according to claim 22, wherein the mask controller 
computes the exposure control signals to maximize the spatial smoothness of the 
exposure values corresponding to the light-sensing elements of the image sensor. 

29. The system according to claim 22, wherein the mask comprises a 
liquid crystal array. 

30. The system according to claim 22 further comprising a first lens 
system located between the scene and the mask for projecting an image of the scene 
onto the mask and a second lens system for projecting the image of the scene as 
transmitted through the mask onto the array of light-sensing elements. 

3 1 . The system according to claim 30 further comprising a light 
difiuser adjacent the mask for removing directionality of light focused onto the mask. 
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32. The system according to claim 22, wherein the array of light- 
sensing elements is a solid state image sensing device. 

33. The system according to claim 32, wherein the solid state image 
sensing device is a charge coupled device light-sensing array. 

34. The system according to claim 32, wherein the solid state image 
sensing device is a CMOS light-sensing array. 

35. A system for high dynamic range imaging comprising: 

an image sensor comprising an array of light-sensing elements, the 
image sensor sensing an image of a scene and providing corresponding pixel values 
representing light intensities impinging on respective ones of the light-sensing 
elements, each light-sensing element having a respective first threshold level so as to 
cause the image sensor to provide a saturated pixel value when the intensity of light 
impinging on the light-sensing element is greater than the respective first threshold 
level, each light-sensing element having a respective second threshold value so as to 
cause the image sensor to provide a blackened pixel value when the intensity of light 
impinging on the light-sensing element is below the respective second threshold level, 
the image sensor having a low dynamic range relative to the range of light intensities 
of the image of the scene, the light-sensing elements having respective spatially 
varying photosensitivies to incident light and corresponding photosensitivity values 
indicative of the respective photosensitivities 

a first memory storing photosensitivity values corresponding to each of 
the light-sensing element; and 

an image processor coupled to the array of light-sensing elements for 
receiving the pixel values provided by the image sensor and coupled to the first 
memory for receiving the photosensitivity values corresponding to the light-sensing 
elements, the image processor comprising a normalizer for mapping the pixel values 
by a function of the photosensitivity values to derive corresponding normalized pixel 
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values, and an interpolator for interpolating the normalized pixel values to derive 
interpolated pixel values at respective positions of a second array overlapping the 
array of light-sensing elements. 

36. The system according to claim 35, wherein the normalizer of the 
image processor maps the pixel values by a function of the photosensitivity values by 
dividing each of the pixel values by the photosensitivity value corresponding to the 
light-sensing element receiving light intensity represented by the pixel value. 

37. The system according to claim 35, wherein the light-sensing 
elements of the image sensor are disposed at respective intersections of a first grid 
defined by spaced intersecting perpendicular lines extending in a vertical direction and 
a horizontal direction, and the positions of the interpolated pixel values are respective 
intersections of a second grid defined by spaced intersecting lines extending in the 
vertical direction and in the horizontal direction. 

38. The system according to claim 37, wherein the interpolator of the 
image processor separately interpolates normalized pixel values in each different 
region of the first grid, each region overlapping adjacent regions and encompassing a 
predetermined number of intersections of the first grid. 

39. The system according to claim 37, wherein the array of light- 
sensing elements of the image sensor comprises repetitively disposed identical groups 
of light-sensing elements having different photosensitivities. 

40. The system according to claim 39, wherein in each of the 
repetitively disposed identical groups of light-sensing elements consists of four 
neighboring light-sensing elements having different photosensitivities, and the first 
and second grids having identical dimensions and are displaced from one another by 
one-half grid position in the horizontal direction and one-half grid position in the 
vertical direction. 



WO 00/79784 



37 



PCT/US00/14515 



41. The system according to claim 38, wherein the interpolator of the 
image processor derives a respective interpolated pixel value in each different region 
of the first grid by taking the average of all normalized pixel values in the region 
which do not correspond to a saturated pixel value or a blackened pixel value. 

42. The system according to claim 35, wherein the image sensor has a 
respective response function to light impinging on each of the light-sensing elements 
and the system further comprises a second memory storing the respective response 
functions of the image sensor, and wherein the image processor is coupled to the 
second memory for receiving the response functions and further comprises a calibrator 
for converting the pixel values provided by the image sensor to linear response pixel 
values in accordance with the respective response functions before the pixel values are 
normalized by the normalizes 

43. The system according to claim 35, further comprising an output 
image memory coupled to the image processor for receiving and storing the 
interpolated pixel values. 

44. A method for high dynamic range imaging comprising the steps of: 
exposing an image sensor comprising an array of light-sensing 

elements to an image of a scene using a spatially varying exposure, the image sensor 
sensing the spatially varying exposure image and providing corresponding pixel 
values representing light intensities impinging on respective ones of the light-sensing 
elements, the image sensor providing a saturated pixel value when the intensity of 
light impinging on a corresponding one of the light-sensing elements is greater than a 
first threshold level and providing a blackened pixel value when the intensity of the 
light impinging on a corresponding one of the light-sensing elements is below a 
second threshold level, the image sensor having a low dynamic range relative to the 
range of light intensities of the image of the scene; and 
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normalizing the pixel values provided by the image sensor with respect 
to the spatially varying exposure of the light-sensing elements to derive corresponding 
normalized pixel values. 

45. The method according to claim 44, wherein the step of exposing 

5 the image sensor using a spatially varying exposure includes the step of using a mask 
having a multiplicity of light transmitting cells each controlling the exposure of a 
respective one or more of the light-sensing elements of the image sensor to light from 
the scene, each of the light-sensing element having a corresponding exposure value 
indicative of the transparency of the mask cell through which light impinging on the 
10 light-sensing element passes, and the step of normalizing the pixel values provided by 
the image sensor includes the step of mapping the pixel values by a function of the 
exposure values to derive corresponding normalized pixel values. 

46. The method accordiing to claim 45, wherein the step of mapping 
the pixel values by a function of the exposure values comprises dividing each of the 

1 5 pixel values with the exposure value corresponding to the light-sensing element 
receiving light intensity represented by the pixel value. 

47. The method according to claim 44, wherein the exposure values 
corresponding to the light-sensing elements are fixed, and further comprising the step 
of interpolating the normalized pixel values to derive interpolated pixel values at 

20 respective positions of a second array overlapping the array of the light-sensing 
elements. 

48. The method according to claim 44, wherein the step of 
interpolating the normalized pixel values include the step of applying an interpolation 
filter to the normalized pixel values. 

25 49. The method according to claim 47, wherein the step of 

interpolating the normalized pixel values includes the steps of separately interpolating 
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normalized pixel values in each different region of the array of the light-sensing 
elements, each region overlapping adjacent regions and encompassing a 
predetermined number of the light-sensing element positions. 

50. The method of claim 49, wherein the steps of separately 

5 interpolating normalized pixel values in each different region of the array of light- 
sensing elements each include the step of taking the average of all normalized pixel 
values in the region, except for ones that correspond to a saturated pixel value or a 
blackened pixel value, to derive a respective one of the interpolated pixel values. 

5 1 . The method of claim 44 further comprising the step of calibrating 
1 0 the pixel values provided by the image sensor according to a response function of the 

image sensor to derive linear response pixel values before the step of normalizing the 
pixel values. 

52. The method according to claim 47 further comprising the step of 
storing the interpolated pixel values in an output image memory. 

15 53. The method of claim 45 further comprising the steps of projecting 

an image of the scene onto the mask, and projecting the image of the scene as 
transmitted through the mask onto the array of light-sensing elements. 

54. The method of claim 53, wherein the step of projecting the image 
of the scene as transmitted through the mask onto the array of light-sensing elements 
20 includes projecting the image of the scene as transmitted through the mask and a light 
diffuser onto the array of light-sensing elements. 



55. The method of claim 45, wherein the exposure value 
corresponding to each light-sensing element is alterable by applying a respective 
exposure control signal to the mask, and further comprising the steps of: 
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computing from the pixel values provided by the image sensor 
exposure control signals for minimizing the number of saturated pixel values and 
blackened pixel values from the image sensor; 

applying the computed exposure control signals to the mask; and 
5 providing exposure values corresponding to the applied exposure 

control signals for use in the normalizing step. 

56. The method according to claim 55 further comprising the step of 
storing the normalized pixel values in an output image memory. 

57. The method according to claim 55 further comprising the steps of: 
1 0 smoothing the exposure values corresponding to the applied exposure 

control signals by applying a smoothing filter to the exposure values to derive 
smoothed exposure values corresponding to the normalized pixel values; 

reapplying the smoothed exposure values to the normalized pixel 
values by multiplying each normalized pixel value with a corresponding smoothed 
1 5 exposure value to derive monitor image pixel values; and 

storing the monitor image pixel values in a monitor image memory. 

58. The method of claim 55, wherein the step of computing exposure 
control signals comprises computing exposure control signals to maximize the signal- 
to-noise ratio of the average pixel value provided by the image sensor. 



20 59. The method according to claim 55, wherein the step of computing 

exposure control signals comprises computing exposure control signal to maximize 
the spatial smoothness of the corresponding exposure values. 

60. A method for high dynamic range imaging comprising the steps of: 
exposing a frame of photographic film to an image of a scene through a 
25 mask having a multiplicity of light transmitting cells controlling the exposures of 

respective regions of the film to light from the scene, each of the respective regions of 
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the film having a corresponding exposure value indicative of the transparency of the 
masked cell through which light impinging on the region passes; 

storing the exposure values corresponding to the respective regions of 

the film; 

processing the film to develop a masked image contained thereon or on 

a print thereof; 

scanning the processed film or a print thereof with an image scanner to 
derive pixel values corresponding to the respective regions of the film or a print 
thereof; and 

normalizing the pixel values from the scanner using the stored 
exposure values to derive normalized pixel values. 

61. The method according to claim 60 further comprising the step of 
interpolating the normalized pixel values to derive interpolated pixel values. 

62. The method according to claim 61 further comprising the step of 
storing the interpolated pixel values in an output image memory. 

63. The method of claim 60 further comprising the step of aligning the 
pixel values corresponding to the respective regions of the film or a print thereof with 
the exposure values corresponding to the respective regions. 

64. The method according to claim 60 further comprising the step of 
calibrating the pixel values corresponding to the respective regions of the film or a 
print thereof in accordance with a response function of the film, the image scanner and 
a printer, if any, to derive linearized pixel values before the step of normalizing the 
pixel values. 
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